Multiple Comparison Procedures for Determining the Optimal Complexity of a Model

We aim to determine which of a set of competing models is better statistically, that is, on average. A way to define “on average” is to consider the performance of these algorithms averaged over all the training sets that might be drawn from the underlying distribution. When comparing more than two means, an ANOVA F-test tells you whether the means are significantly different from each other, but it does not tell you which means differ from each other. A simple approach is to test each possible difference by a paired ttest. However, the probability of making at least one type I error increases with the number of tests made. Much research has been done over the years to find ways around these problems. The resulting techniques are known as multiple comparison procedures. We briefly discuss these methods and comment its potential advantages. Finally, we show how to apply a well known multiple comparison procedure (Bonferroni method) to model selection by determining the optimal degree in polynomial fitting and the optimal number of hidden neurons in feedforward neural networks.